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During  the  period  of  this  report,  we  completed  construction  of  what  we  believe  to  be  the 
first  neural  network  of  biology-like  pulsating  neuron.  The  network  consists  of  N=32  PUTONs 
(programmable  unijunction  transistor  neurons).  The  coupling  between  neurons  was  realized  by 
way  of  a  JPL  programmable  synaptic  weight  chip.  Methods  for  monitoring  the  collective  firing 
activity  and  characterizing  the  spiking  behavior  of  neurons  in  the  network  were  also  developed.  A 
simplified  circuit  diagram  of  the  network,  with  only  one  PUTON  shown,  is  given  in  Fig.  1.  The 
mode  of  coupling  between  neuron  and  mutual  control  of  firing  activity  in  this  first  embodiment  is 
via  modulation  of  the  grid  voltage  of  the  PUTON  by  the  net  input  from  its  presynaptic  neurons. 
Other  modes  of  coupling,  e.g.  by  modulating  the  extinction  voltage  will  be  studied  in  the  future. 

To  date,  we  find  that  when  the  connection  weights  matrix  is  selected  randomly,  the 
network  cannot  reach  phase-lock.  However  any  structuring  in  the  connectivity  matrix  results 
usually  in  phase-locking  manifested  by  ordered  firing  of  the  neurons.  The  network  is  capable 
therefore  of  self-organization  which  makes  it  potentially  capable  of  carrying  out  optimization  and 
learning  tasks.  We  have  also  verified  that  when  phase-lock  is  established,  the  net  synaptic  input  to 
each  neuron  is  periodic  (see  Fig.  2).  This  justifies  our  earlier  use  of  periodic  (cosinusoidal)  signals 
to  characterize  the  behavior  of  a  single  isolated  PUTON  by  bifurcation  diagrams,  that  illustrate  the 
functional  complexity  of  these  processing  units. 

One  important  aspect  of  exploring  the  dynamics  of  the  pulsating  network  is  that  it 
demonstrates  potential  for  solving  temporal  optimization  problems  whose  solutions  involve  a 
temporal  ordering  of  events;  a  good  example  is  the  N-city  Traveling  Salesman  Problem  (TSP-N). 
Spiking  networks  in  this  regard  have  an  advantage  over  embedding  and  solving  TSP-N  in 
networks  of  sigmoidal  neurons  in  that  only  N  spiking  neurons  are  needed  instead  of  the  usual  N^ 
sigmoidal  neuron.  This  can  be  significant  advantage  when  N  is  large.  Furthermore,  the  speed 
advantage  of  solving  the  TSP-N  with  analog  hardware  over  numerical  (digital)  solutions  is 
preserved  in  spiking  network.  This  speed  advantage  becomes  significant  as  N  is  increased, 
because  while  solution  time  in  digital  hardware  grows  exponentially  with  N,  the  time  needed  to 
reach  phase-lock  in  our  pulsating  network  does  not  change  significantly  with  N  (N  was  changed 
from  N=5  to  its  maximum  value  in  our  network  Nm^x  =  32). 

Several  techniques  for  inducing  meaningful  phase-lock  states  in  our  network  were  studied. 
One  simple  method  of  setting  initial  conditions  for  the  network  that  we  used,  is  to  temporarily 
reduce  the  supply  voltage  of  the  PUTONs  to  a  low  value,  such  that  none  of  the  neurons  can  fire, 
and  then  gradually  increase  the  supply  voltage  back  to  its  original  value.  In  many  cases,  the 
network  will  reach  another  steady-state  (phase-lock  state),  which  will  sometimes  correspond  to  a 
better  solution.  Another,  is  a  method  for  annealing  the  network  by  temporarily  removing  one  of 


the  neurons  from  the  network,  and  then  re-inserting  it.  This  is  accomplished  by  temporarily  setting 
the  weights  between  a  specified  neuron  and  each  of  the  remaining  neurons  to  zero,  and  then 
returning  them  to  their  original  values. 

A  way  of  allowing  the  system  to  develop  in  a  specific  order  is  to  include  only  two  neurons 
in  the  network  in  the  beginning,  and  then  gradually  include  one  by  one  other  neurons  into  the 
network.  This  procedure  can  be  realized  by  setting  the  weights  to  zero  for  those  neurons  not 
included  in  the  network,  and  then  gradually  changing  the  relevant  weights  to  the  desired  values. 
The  order  in  which  the  neurons  are  added  can  however  influence  the  steady-state  solutions  of  the 
system. 

In  our  experiments  on  the  network,  we  focused  on  the  phase-locking  phenomenon  of  the 
network  as  a  possible  mechanism  for  solving  optimization  problems.  There  are,  however,  other 
dynamics  of  the  network  that  can  be  exploited.  From  the  experiments  performed  on  single 
neurons,  we  know  that  our  neuron  models  can  exhibit  chaotic  dynamics.  Therefore,  there  are 
other  possible  applications  of  such  behavior  to  be  explored.  Further  work  in  studying  the  network 
of  pulsed  neurons  is  to  build  networks  that  implement  other  mechanisms  of  coupling  among  the 
neurons.  It  can  be  expected  that  such  networks  will  exhibit  more  complicated  dynamics  than  those 
exhibited  so  far  by  our  current  network. 

One  goal  of  our  work  with  the  spiking  network  is  to  gain  insight  into  how  the  order  of 
firing  of  neurons  can  be  controlled  by  the  weights  matrix  and  initial  conditions.  This  facet  of  the 
work  is  important  because  it  is  related  to  the  more  challenging  task  of  finding  a  rule  or  algorithm 
for  learning  in  pulsating  neural  networks.  A  learning  rule  for  pulsating  neural  network  would  have 
far  reaching  implications  because  it  implies  ability  to  learn,  recognize,  or  generate  spatio-temporal 
patterns  more  naturally  and  direcdy  than  in  the  limited  methods  developed  for  sigmoidal  networks. 

During  the  period  of  this  report,  two  reports  of  our  research  under  this  grant  were 
published: 

1 .  Z.  Wen,  A.  Baek,  and  N.H.  Farhat,  "Optoelectronic  neural  dendritic  tree  processing 
with  electron-trapping  materials",  Optics  Letters,  vol.  20,  pp.  614-616,  March,  1995. 

1.  J.R.  Tower  and  N.  Farhat,  "The  transversal  imager:  a  photonic  neurochip  with 
programmable  synaptic  weights",  IEEE  Trans,  on  Neural  Networks,  vol.  6,  pp.  248- 
251,  Jan.  1995. 

Copies  of  these  publications  are  included  with  this  report. 
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Figure  2:  Two  typical  waveforms  of  the  network,  with  symmetric  random  weights 

between  0  and  -63.  Channel  1:  output  pulses  Voi  of  neuron  1.  Channel  2:  output  pulses 
Vo2  of  neuron  2.  Channel  3:  gate  voltage  Vgi  of  neuron  1.  Channel  4:  membrane  potential 
Vmi  of  neuron  1. 
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We  show  that  under  simultaneous  illumination  of  pulsed  blue  light  and  constant  IR  light  the  dynamic  responses  of 
electron-trapping  materials  could  be  employed  to  implement  optically  controlled  neural  dendritic  responses.  The 
importance  of  neurocomputing  with  biology-oriented  spiking  neurons  and  the  role  played  by  dendritic  trees  are 
discussed.  Computer  simulations  of  dendritic  responses  in  biological  neuron  and  experimental  results  of  electron- 
trapping  material  dynamics  are  presented.  These  results  show  that  electron-trapping  materials  are  well  suited 
for  implementing  optically  controlled  dendritic  responses  for  use  in  large-scale  biology-oriented  optoelectronic 
spiking  neural  networks. 


Electroii’trapping  materials  (ETM’s)  have  received 
much  attention  in  optical  information  processing, 
erasable  three-dimensional  optical  memory,  and  op¬ 
tical  neural  networks^  because  of  their  many  attrac¬ 
tive  properties,  such  as  large  linear  dynamic  range, 
high  usable  resolution,  and  fast  response  times.  Re¬ 
cently  we  initiated  a  study  of  the  dynamics  of  ETM’s 
under  sumultaneous  illumination  of  blue  light  and 
IR  light^  and  uncovered  a  new  set  of  dynamics  that 
makes  ETM’s  uniquely  suited  for  use  in  optoelec¬ 
tronic  implementation  of  large-scale  biology-oriented 
spiking  (or  pulsating)  neural  networks.  In  a  pre¬ 
vious  Letter  we  demonstrated  the  utility  of  ETM’s 
in  producing  dense  arrays  of  optoelectronic  spiking 
neurons.'^  In  this  Letter  we  describe  the  use  of 
ETM’s  in  producing  optically  controlled  dendritic 
responses  for  optoelectronic  spiking  neural  networks. 
The  importance  of  neurocomputing  with  spiking 
neurons,  in  particular  the  role  played  by  the  neu¬ 
rons’  dendrites,  is  first  discussed.  Then  computer 
simulation  results  of  dendritic  responses  in  biological 
neuron  and  experimental  results  of  ETM  dynamics 
are  presented.  It  is  shown  that  ETM’s  are  well 
suited  for  implementing  optically  controlled  dendritic 
responses,  i.e.,  postsynaptic  responses  or  potentials, 
of  the  kind  needed  in  optoelectronic  implementa¬ 
tion  of  large-scale  biology-oriented  pulsating  neural 
networks. 

In  contrast  to  most  conventional  neural  network 
models  studied  today,  which  assume  that  computa¬ 
tional  power  stems  from  the  massive  interconnec¬ 
tions  (synaptic  weights)  among  the  large  numbers  of 
functionally  simple  processing  elements  (sigmoidal  or 
binary  neurons)  and  which  ignore  explicitly  the  spik¬ 
ing  nature  of  biological  neurons  as  well  as  the  role 
of  relative  timing  between  action  potentials,  biologi¬ 
cal  neurons  are  both  functionally  and  structurally 
complex  processing  elements.  It  is  reasonable  to  ex¬ 
pect  that  this  functional  complexity  would  reflect  it¬ 
self  in  the  higher-level  computational  power  of  the 
networks  they  form,  and  therefore  development  of 
artificial  neuron  models  that  emulate  the  functional 
complexity  of  biological  neurons  would  lead  to  a  new 
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generation  of  neural  networks  with  vastly  enhanced 
processing  power.  Recent  physiological  experimen¬ 
tal  results  strongly  support  these  conjectures  (see, 
e.g.,  Refs.  6  and  7).  The  functional  complexity  of 
biological  neurons  stems  from  two  properties:  (a) 
the  complex  dynamics  of  the  excitable  biological  mem¬ 
brane  and  (b)  the  preprocessing  conducted  by  den¬ 
drites  on  incident  spike  trains  from  other  neurons. 
Our  study  in  modeling  the  excitable  biological  mem¬ 
brane,  based  on  a  limit-cycle  oscillator  emplo3dng  an 
S-shaped  nonlinearity  that  emulates  the  excitable 
biological  membrane  to  form  what  we  call  bifurca¬ 
tion  neuron,  shows  that  the  bifurcation  neuron  model 
preserves  much  of  the  functional  complexity  of  bio¬ 
logical  neurons  and,  particularly  under  periodic  ac¬ 
tivation  potential,  exhibits  complex  firing  modality 
such  as  integer  harmonic  and  subharmonic  phase- 
locked  firing,  period-m  firing,  quasi-periodic  firing, 
and  period-doubling  routes  to  chaos,  depending  on 
the  parameters  of  its  periodic  activation.®  A  cru¬ 
cial  assumption  in  the  above  study  is  that  dendritic 
tree  processing  produces  periodic  activation  poten¬ 
tial  when  the  impinging  incident  spike  trains  are  co¬ 
herent.  Such  coherent  incident  spike  activity  occurs 
whenever  a  network  of  spiking  neurons  phase  locks. 
Therefore  dendritic  tree  processing  plays  a  central 
role  in  spiking  neural  networks  in  their  abilities  to 
phase  lock  and  to  exhibit  rich  functional  complexity. 

In  biological  neural  networks,  the  arrival  of  an  ac¬ 
tion  potential  at  a  synapse  site  on  the  dendrite  of 
a  target  neuron  triggers  a  chain  of  electrochemical 
events.  The  result  is  the  production  of  a  postsynap¬ 
tic  potential  (PSP)  that  alters  the  activation  potential 
(membrane  potential)  at  the  target  neuron’s  hillock, 
which  is  the  trigger  zone  for  action  potential  gen¬ 
eration.  The  polarity,  amplitude,  and  shape  of  the 
PSP  waveform  depend  on  the  type  of  neurotrans¬ 
mitter  involved  in  synaptic  transmission,  on  the  lo¬ 
cation  of  the  synapse  on  the  dendritic  tree,  and  on 
the  propagation  from  the  synapse  to  the  hillock.  In 
an  actual  situation,  the  activation  potential  at  the 
hillock  of  a  neuron  is  the  superposition  of  a  large 
number  of  such  PSP’s  produced  by  the  barrage  of  ac- 
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(a) 


(b) 


Fig.  1.  (a)  Diagram  of  a  model  neuron  with  a  single 

dendrite  with  10  excitatory  and  10  inhibitory  synapses 
used  in  computer  simulations  of  PSP’s  formed  at  the 
hillock  by  injection  of  current  pulses  at  selected  synapses, 
(b)  PSP’s  generated  by  a  transient  diffusion  differen¬ 
tial  equation  of  a  passive  dendrite  for  three  excitatory 
synapse  positions  of  injected  current  pulses.  Membrane 
resistance,  50  H  cm^;  membrane  capacitance,  1  ^F/cm^; 
intracellular  resistance,  100  Cl  cm  (see  Ref.  9). 


ron’s  hillock  to  produce  periodic  activation  potential. 
Furthermore,  such  periodic  activation  is  sufficiently 
robust  with  fluctuations  in  synaptic  transmission  ef¬ 
ficacy  and,  to  a  lesser  extent,  in  synaptic  delay  time. 

We  now  show  that  under  simultaneous  illumina¬ 
tion  of  pulsed  blue  light  and  constant  IR  light  the 
ETM  responses  can  be  used  to  implement  optically 
controlled  dendritic  responses.  We  model  ETM  dy¬ 
namics  by  the  following  equations^: 

=  alait)  ”  (3n{t)Im(t),  (1) 

dr 

Io{t)  =  Klsit)  +  yn{t)IiB.{t) ,  (2) 

where  n(t)  denotes  the  trapped  electron  density;  h  it), 
Imit),  and  loit)  represent  the  blue  light,  the  IR  light, 
and  the  emitted  orange  light  intensities;  a  and  k 
are  coefficients  for  trap  creation  and  fluorescence  by 
the  blue  light;  and  y  and  (3  are  coefficients  for  IR- 
stimulated  emission  and  trap  erasing  by  the  IR  light, 
respectively.  The  dynamics  of  interest  here  is  iden¬ 
tical  to  the  situation  considered  in  Ref.  5,  i.e.,  Im  is 
static  and  I  nit)  consists  of  a  static  term  and  a  pulse- 
modulated  term  as  follows: 

hit)  =  h  +  h  ? [uit  -  kT)  -  u{t  -kT  -  tj],  (3) 

k=0 

where  u{t)  denotes  the  unit  step  function,  h  and  h 
are  the  static  and  pulsed  blue  light  intensities,  and  T 
and  are  the  pulse  period  and  the  pulse  width,  re¬ 
spectively.  The  solution  for  the  emitted  orange  light 
loit)  during  the  iVth  period  is^ 


fo(^) 


K{h  +  fi)  ^ - ^  +  oLjliIm 


kIo  H - -  +  aylilinta 


[exp(/3/iRT)  -  1] 
exp[-~ /3I]R(t  -  NT  -  ^„.)] 
[1  -  exp(-/3/iRr)] 


t-NT\ 


for  NT  <t<  NT  + 

for  NT  +  <  t  <  {N  +  1)T 


(4) 


tion  potentials  from  other  neurons  arriving  at  any 
time  at  the  synapse  sites  on  its  dendrites.  The  im¬ 
portance  of  dendritic  tree  processing  was  studied 
recently.^’^®  By  assuming  the  dendrite  to  be  passive, 
it  is  possible  for  one  to  derive  a  transient  diffusion 
differential  equation  (the  cable  equation)  and  to  de¬ 
termine  the  response  of  a  dendrite  to  current  pulses 
injected  at  different  positions  along  its  length.^  In¬ 
jected  current  pulses  represent  the  effect  of  arriving 
action  potentials.  Using  commonly  listed  dendritic 
parameters,^  we  have  carried  out  such  a  computation 
for  the  simplified  arrangement  shown  in  Fig.  1(a)  for 
current  pulses  of  uniform  amplitude  injected  at  three 
synapses  along  the  dendrite.  The  results,  given  in 
Fig.  1(b),  show  how  the  shape  of  the  PSP  and  its 
rise  time,  peak  value,  latency  (peak  position  ),  and  de¬ 
cay  time  all  depend  on  synapse  position  (i.e.,  position 
of  injected  current).  In  reality,  the  peak  value  also 
depends  on  the  S3niaptic  efficacy.  We  also  carried 
out  simulations  that  confirm  that  when  the  incident 
spike  trains  are  correlated,  the  smoothing  responses 
of  dendrites  lead  to  PSP’s  that  sum  at  the  neu- 


From  Eq.  (4)  we  observe  that  in  the  emitted  orange 
light  the  contributions  that  are  due  to  IR-stimulated 
emission  and  fluorescence  coexist.  Fluorescence  is 
manifested  by  sudden  jumps  at  the  onset  and  termi¬ 
nation  of  the  blue  light  pulses.  On  the  other  hand, 
IR-stimulated  emission  first  undergoes  a  linear  in¬ 
crease  during  the  blue  light  pulse  in  the  charging 
process  and  then  decays  exponentially  after  the  pulse 
in  the  discharging  process.  An  experiment  has  been 
carried  out  to  study  ETM  dynamics  in  the  above 
situation  by  using  the  same  experimental  setup  used 
in  Ref.  5.  Figure  2  shows  the  experimental  results. 
In  all  these  plots,  h  =  3.98  mW/cm^,  =  25  ms,  and 
T  =  250  ms.  Figure  2(a)  shows  the  ETM  responses 
for  three  different  levels  of  the  static  blue  light  in¬ 
tensity  /o,  whereas  the  IR  light  intensity  Iir  is  con¬ 
stant.  In  Figs.  2(b)  and  2(c),  Iq  remains  unchanged, 
whereas  Iir  is  varied.  From  Fig.  2  one  observes  the 
fluorescence  jumps  at  the  blue  light  pulse  edges,  lin¬ 
ear  buildup  during  the  pulse,  and  trailing  exponential 
decay  after  it,  as  predicted  by  Eq.  (4).  Equation  (4) 
indicates  that  varying  h  shifts  the  ETM  response  up 
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Fig.  2.  Oscilloscope  displays  of  ETM  responses.  Upper 
trace:  Pulsing  signal  from  the  programmable  liquid- 
crystal  beam  gate  (see  Fig.  1  of  Ref.  5).  Lower  trace: 
ETM  responses,  (a)  /ir  =  403  mW/cm^,  /o  =  4.8  ^tW/ 
cm^  (lower  curve),  0.22  mW/cm^  (middle  curve),  and 
0.54  mW/cm^  (upper  curve),  (b)  /ir  =  128  mW/cm“, 
/o  ==  29.4  yuW/cm^.  (c)  /ir  =  372  mW/cm^  h  = 
29.4  /xW/cm^. 

and  down,  and  the  results  shown  in  Fig.  2(a)  gen¬ 
erally  agree  with  this.  The  results  in  Fig.  2  show, 
however,  that  the  heights  of  the  fluorescence  jumps, 
as  well  as  the  magnitude  of  the  exponential  portion 
of  ETM  response,  vary  with  /ir  and  /o  used.  Fur¬ 
thermore,  it  appears  that  the  upward  jump  and  the 
downward  jump  have  different  heights.  The  first- 
order  linear  model  of  ETM  dynamics  given  in  Eqs.  (1) 
and  (2)  fails  to  predict  these  subtle  effects  because 
it  neglects  nonlinear  effects  such  as  trapped  elec¬ 
tron  density  saturation  and  dependence  of  electron¬ 
trapping  efficiency  on  existing  trapped  electron 
density.  Clearly,  to  utihze  ETM’s  effectively  in  ap¬ 
plications  that  make  use  of  their  unique  dynamics 
it  is  necessary  to  develop  an  improved  theoretical 
model  that  would  take  into  account  the  neglected 
effects  mentioned  above.  We  are  in  the  process  of 
developing  such  a  model  and  verifying  it  experi¬ 
mentally;  the  results  will  be  reported  elsewhere. 
Nonetheless,  we  draw  attention  now  to  the  similarity 
between  the  PSFs  in  Fig.  1,  which  represent  bio¬ 


logical  neurons,  and  the  ETM  responses  in  Fig.  2. 
By  varying  the  values  of  Iq  and/or  /ir,  one  can  con¬ 
trol  the  ETM  response  in  various  ways,  e.g.,  up  and 
down  shifts,  different  heights  of  fluorescence  jumps 
and  magnitude  of  exponential  response,  and  faster 
or  slower  exponential  decays.  So  we  have  a  unique 
method  for  crudely  emulating  dendritic  response  in 
biological  neurons  in  a  fashion  useful  for  realizing 
a  dense  optically  controlled  dendritic  response  array 
(or  two-dimensional  dendritic  trees)  for  use  in  large- 
scale  biology-oriented  optoelectronic  spiking  neural 
networks. 

The  spiking  nature  of  biology-oriented  neural  net¬ 
works  makes  studying  their  behavior  by  computer 
simulation  impractical  because  of  the  lengthy  com¬ 
puting  time  involved.  In  a  recent  paper  Prange  and 
Klar.^^  argued  convincingly  that  the  best  way  of 
studying  and  realizing  biology-oriented  neural  net¬ 
works  is  through  analog  complementary  metal-oxide 
semiconductor  technology  rather  than  digital  hard¬ 
ware.  However,  they  showed  that  the  number  of 
neurons  one  can  accommodate  on  a  VSLI  chip  is  lim¬ 
ited  to  a  hundred  or  so,  even  when  submicrometer 
complementary  metal-oxide  semiconductor  technol¬ 
ogy  is  used,  because  of  the  relatively  large  size  of 
the  neuron/ dendrite  cell.  By  combining  ETM's  with 
state-of-the  art  technologies  such  as  smart  pixel  spa¬ 
tial  light  modulators  and  image  intensifiers  to  form 
dense  arrays  of  spiking  neurons  and  dendritic  re¬ 
sponses,  it  is  feasible  that  large-scale  optoelectronic 
spiking  neural  networks  of  10^ neurons  can  be 
realized.  Such  networks  would  be  invaluable  tools 
for  modeling,  studying,  and  demonstrating  the  role  of 
synchronicity,  bifurcation,  and  chaos  in  higher-level 
cortical  functions  and  exploring  their  use  in  enhanc¬ 
ing  the  performance  of  artificial  neural  systems. 

This  research  was  supported  by  the  U.S.  Office  of 
Naval  Research.  The  ETM  film  used  in  this  study 
was  furnished  by  the  Quantex  Corporation,  Rockville, 
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The  Transversal  Imager:  A  Photonic  Neurochip 
with  Programmable  Synaptic  Weights 

J.  R.  Tower  and  N,  Farhat 


Abstract —  A  photonic  neural  processor  implemented  in 
NMOS/CCD  integrated  circuit  technology  is  described.  The 
processor  performs  outer-product  processing  utilizing  optical 
input  of  the  synaptic  weights  and  electrical  input  of  the  state 
vector,  or,  visa  versa.  The  performance  of  the  32-neuron, 
1024-synapse  processor  is  presented. 

1.  Introduction 

HIS  paper  describes  the  design,  fabrication,  and  per¬ 
formance  of  an  NMOS/CCD  integrated  circuit  which 
performs  outer-product,  neural  network  computations.  The 
transversal  imager  integrated  circuit  utilizes  a  unique  architec¬ 
ture  which  permits  (1)  the  synaptic  weights  to  be  introduced 
electrically  to  the  complete  neural  network  in  parallel,  and 
the  one-dimensional  state  vector  to  be  introduced  optically, 
or  (2)  the  synaptic  weights  to  be  introduced  optically  to  the 
complete  neural  network  in  parallel,  and  the  one  dimensional 
state  vector  to  be  introduced  electrically.  In  both  cases,  the 
computation  of  the  outer-product  between  the  two-dimensional 
synaptic  weight  matrix  and  the  one-dimensional  state  vector 
is  performed  simultaneously  across  the  integrated  circuit. 

The  optical  input  to  this  electro-optical  neural  network 
can  thus  be  reduced  to  the  one-dimensional  state  vector. 
This  simplification  of  the  optical  input  requirements  from  a 
two-dimensional  pattern  to  a  one  dimensional  pattern  has  sig¬ 
nificant  impact  on  system  hardware  performance,  and  system 
computational  throughput. 

II.  Description  of  Processing  Algorithm 

As  background  for  understanding  the  transversal  imager 
architecture,  it  is  useful  to  discuss  the  motivation  for  the  archi¬ 
tecture,  namely,  the  efficient  implementation  of  the  Hopfield 
neural  network.  In  the  basic  Hopfield  neural  network  [1],  there 
are  N  neurons  which  are  connected  to  each  other  by  the 
synapses.  The  state  of  the  network  is  given  by  the  vector  V : 

V  =  (VuV2r--Vjv). 

The  network  interconnections  are  designated  by  the  synaptic 
interaction  matrix  W,  whereby  the  weight  Wij  is  the  strength 
of  the  interaction  from  the  jth  neuron  to  the  ith  neuron.  The 
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synaptic  matrix  is  of  dimension  N  x  N.  The  state  of  network 
is  updated  according  to  (1)  the  vector-matrix  multiplication: 

Ii  =  ^W,jV, 

and  (2)  a  transfer  function: 

Vi  =  f{Ii) 

where  Vi  is  the  updated  state  of  the  ith  neuron.  The  state 
of  the  neuron  is  either  excited  (Vi  —  1)  or  not  excited 
(Vi  =  0).  The  interconnection  weights  are  either  unipolar 
binary  [W£j£0(off), +1  (excite)],  bipolar  binary  [WijS  -  1 
(inhibit),  +l(excite)],  ternary  [W^ye  ~  1  (inhibit),  0  (off),  +1 
(excite)],  or  analog.  The  transfer  function  is  typically  a  sharp 
thresholding  function. 

III.  Prior  CCD  Neural  Network  Approaches 

All  of  the  charge  coupled  device  neural  processing  inte¬ 
grated  circuits  reported  to  date  (1)  exploit  the  analog,  rather 
than  digital,  capability  of  CCD’s,  and  (2)  use  charge  domain 
processing  to  reduce  the  area  required  for  interconnections 
and  computational  circuitry.  In  the  work  reported  by  Sage  et 
al  of  MIT  Lincoln  Laboratories  [2],  surface  channel  CCD 
technology  and  metal  nitride  oxide  semiconductor  (MNOS) 
technology  is  combined  to  produce  circuits  capable  of  outer 
product  processing.  In  this  approach,  state  vectors  are  in¬ 
troduced  electrically,  and  the  analog  synaptic  weights  are 
produced  electrically  and  stored  in  MNOS  gates.  The  MNOS- 
CCD  gates  function  to  store  the  synaptic  weight  and  to  control 
the  flow  of  charge,  thereby  producing  the  product  of  the  input 
state  and  the  stored  weight.  The  sum  of  products  is  produced  in 
the  charge  domain  and  then  converted  to  the  voltage  domain 
to  produce,  in  a  Hopfield  net  application,  the  desired  outer- 
product  result. 

In  the  CCD  neural  network  processor  work  by  Chiang  et  al. 
of  MIT  Lincoln  Laboratory  [3],  buried  channel  CCD  structures 
are  utilized  for  a  variety  of  functions  including  the  implemen¬ 
tation  of  multiplying  digital-to-analog  converters  (MDAC’s). 
In  this  architecture,  when  applied  to  the  Hopfield  algorithm, 
the  state  vector  is  introduced  electrically  to  an  analog  CCD 
register.  The  synaptic  weights  are  introduced  electrically  to 
6-bit  CCD  digital  memories.  The  MDAC’s  perform  the  outer- 
product  multiplication  of  the  analog  state  vector  and  the  6-bit 
digital  synaptic  weights,  with  the  summation  of  products  being 
achieved  in  the  charge  domain.  The  output  charge  packets  are 
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then  converted  into  the  voltage  domain  to  produce  the  final 
outer  product  result. 

The  CCD  neural  processor  which  is  most  similar  to  the 
transversal  imager  is  that  reported  by  Agranat,  et  al.  of  the 
California  Institute  of  Technology  [4].  In  this  work  a  Hopfield 
network  is  realized  utilizing  buried  channel  CCD  technology. 
The  state  vector  input  is  binary  (+1,0)  and  is  introduced 
electrically.  The  synaptic  weights  are  input  electrically  or 
optically,  and  are  analog.  The  outer  product  multiplication 
of  the  binary  state  vector  and  each  analog  synaptic  weight 
is  performed  by  controlled  (+1  charge  flow,  0  no  charge 
flow)  gating  of  replicas  of  the  recirculating  synaptic  weights. 
The  gated  products  are  added  together  in  the  charge  domain, 
forming  the  outer  product  sum-of-products.  The  outer  product 
terms  are  then  serially  read  out  in  the  charge  domain  to 
an  output  where  the  final  voltage  output  is  produced.  This 
architecture  therefore  utilizes  a  two-dimensional  optical  input 
to  introduce  the  analog  synaptic  weights,  and  the  binary  state 
vector  is  introduced  electrically. 


IV.  Transversal  Imager  Architecture 

The  transversal  imager  concept  grew  out  of  work  con¬ 
ducted  in  the  late-1970’s  and  early  1980’s  in  the  field  of 
programmable  CCD  analog-binary  transversal  filters  [5],  [6]. 
The  transversal  imager  architecture  we  use  is  shown  in  Fig.  1. 
This  signal  processing  imager  [7],  [8]  was  designed  to  per¬ 
form  vector-matrix  multiplication  by  (1)  imaging  the  one¬ 
dimensional  input  state  vector  (V)  spread  across  the  rows 
of  the  X  V  image  area,  (2)  performing  the  interconnect, 
synaptic  weighting  (W)  at  each  pixel  location  in  parallel, 
(3)  performing  the  summation  along  each  column  in  parallel, 
and  (4)  reading  out  the  sum-of-products  terms  serially.  The 
architecture  was  chosen  to  realize  the  following  desirable 
features: 

1)  Optical  input  of  the  one-dimensional  state  vector. 

2)  Electrical  input  of  the  matrix  weighting  at  each  vector- 
matrix  multiplication. 


Weights  (Analog)  Transversal  Imager 


N  Element  Vector 
of  Electrical  Weights 
For  Input  of  State 
Vector  (Binary) 

Nz:32 


N  Element 
Outer  Product 
Output 


(b) 


Fig.  2.  (a)  Transversal  imager  configured  for  optical  input  of  state  vector, 

(b)  Transversal  imager  configured  for  optical  input  of  synaptic  weights. 


3)  Simultaneous  ternary  weighting  (-1,0,+!)  at  each 
pixel. 

4)  High  vector-matrix  multiplication  frame  rate;  with  frame 
rate  dominated  by  serial  output  register  readout  rate. 

5)  High  optical  integration  time  duty-cycle;  with  integration 
time  a  very  high  percentage  of  frame  rate. 

Each  pixel  of  the  transversal  imager  consists  of  two  photo- 
capacitors  which  see  the  same  optical  input,  two  stages  of 
an  NMOS  digital  shift  register,  control  gates,  and  summation 
busses  (y]+and^-).  The  ternary  weighting  is  achieved  by 
controlling  the  transfer  of  the  photocharge  integrated  in  the  two 
photo-capacitors.  If  both  photo-capacitors  contribute  charge 
to  the  summation  plus  bus  the  pixel  weight  is  +1,  if 
both  contribute  to  the  summation  minus  (I]“)  bus  the  pixel 
weight  is  —1,  and  if  one  contributes  to  the  while  the 
other  contributes  to  the  the  weight  is  0. 

The  NMOS  digital  shift  registers  control  the  selective  trans¬ 
fer  of  photocharge  to  the  bus  based  on  the  data  in  the  shift 
register.  Once  the  selective  transfer  to  the  column  busses 
is  completed,  all  of  the  photo-capacitors  transfer  the  remaining 
charge  to  the  respective  column  busses.  In  this  fashion 
the  imaged  photocharge  is  weighted  in  parallel  by  selective 
charge  transfer. 

The  column  ^^and  busses  receive  the  photocharge 
and  perform  a  nearly  instantaneous  summation  of  the  column 
products  in  the  charge  domain.  The  weighted  and  summed 
photocharge  from  the  N  columns  are  pulled  off  of  the  TV— 
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Cycle  3  optical  integration 


Fig.  3.  Transversal  imager  operating  cycle. 

and  busses  and  transferred  into  a  serial  output  register 

by  charge  primed  couplers  [9],  The  serial  output  register 
switches  the  and  Y~  signals  to  respective  floating 
diffusion  output  amplifiers  after  appropriate  stage  delays.  The 
and  Y~  output  amplifiers  are  differentially  sensed  to 
produce  the  desired  outer  product  result. 

The  primary  operating  configuration  is  shown  schematically 
in  Fig.  2(a).  The  optical  input  of  the  state  vector  is  provided  by 
a  display,  or  more  commonly  an  N  element  LED  array  and 
anamorphic  optics,  which  spreads  the  light  across  the  rows 
of  the  transversal  imager.  The  state  vector  may  be  analog, 
but  is  usually  confined  to  a  binary  on  or  off.  Each  of  the  N 
(LED)  elements  illuminates  all  of  the  photo-capacitor  pairs 
in  it’s  associated  row.  The  N  x  2N  elements  of  the  ternary 
synaptic  weight  matrix  are  introduced  through  multiple  parallel 
electrical  input  ports. 

An  alternate  operating  configuration  is  shown  schematically 
in  Fig.  2(b).  In  this  configuration,  the  synaptic  weights  are 
input  optically.  A  display  with  N  x  2N  resolution  elements 
or  a  spatial  light  modulator  with  at  least  N  x  2N  elements 
projects  the  weights.  The  state  vector  is  introduced  electrically 
with  either  unipolar  binary,  bipolar  binary,  or  ternary  weights. 
The  power  of  this  approach  is  that  bipolar  analog  synaptic 
weights  may  be  achieved  with  this  configuration.  Thus,  a 
continuous  range  of  both  inhibitory  and  excitatory  weights 
may  be  introduced  to  the  outer  product. 

V.  Operating  Cycle 

To  maximize  the  speed  of  the  transversal  imager  the  device 
was  designed  to  perform  simultaneous  operations.  The  device 
operating  cycle  starts  with  the  initiation  of  the  electrical 
weight  read-in  through  multiple  digital  input  ports  (Tl).  At 
the  completion  of  the  operation,  the  weighting  and  summation 
is  performed  by  selective  transfer  of  the  photocharge  (T2). 
The  sum-of-product  terms  are  then  transferred  into  the  serial 
output  register  (T3).  This  is  followed  by  serial  readout  and 
development  of  the  final  outer-product  terms  (T4). 

The  optical  input  and  the  integration  of  the  photocharge  may 
occur  continually.  The  only  time  period  where  erroneous  signal 
can  be  produced  is  during  the  selective  transfer  (weighting) 
process.  This  time  period  is  a  very  small  portion  of  the 
complete  cycle  time.  Therefore,  a  choice  can  be  made  between 
either  a  small  error  in  the  signal  with  the  optical  input  always 
present  or  an  operational  procedure  whereby  the  optical  input 
is  turned  off  during  the  selective  transfer  process. 

The  high  optical  duty  cycle  and  the  rapid  weighting  oper¬ 
ation  are  complemented  by  a  design  which  perimts  a  rapid 


transfer  into  the  serial  output  register.  These  characteristics 
contribute  to  a  high  sensitivity  and  a  total  latency  which  is 
dominated  by  the  serial  output  rate. 

To  assure  that  the  weight  introduction  process  is  not  ad¬ 
versely  impacting  the  transversal  imager  cycle  time,  the  digital 
register  design  incorporates  multiple  input  ports  and  a  digital 
clock  rate  such  that  the  weight  introduction  time  (Tl)  is  less 
than  the  serial  readout  time  (T4).  Furthermore,  the  weight- 
ing/summation  process  (T2)  and  the  transfer  into  the  serial 
register  (T3)  are  rapid  compared  to  the  serial  readout  time 
(r4).  Thus,  the  device  may  be  ran  in  the  manner  shown  in 
Fig.  3,  assuring  a  cycle  time  essentially  equal  to  the  serial 
output  register  readout  time  (T4). 

An  approximation  of  analog  synaptic  weights  can  be 
achieved  with  this  processor  architecture.  By  performing 
the  above  described  operating  cycle  on  a  series  of  synaptic 
weight  “bit  planes,”  accumulating  the  outer  product  terms,  and 
then  applying  the  appropriate  1/2”  weighting,  the  synaptic 
weights  can  be  implemented  to  u-bits.  The  penalty  of  this 
increased  resolution  in  the  weights  is  the  increase  from  a 
single  computational  cycle  to  n  cycles. 

VI.  Performance  Results 

A  32  X  32  transversal  imager  has  been  fabricated  in  a  3- 
^m  NMOS/  buried  channel  CCD  process.  The  device  has  a 
square  200  x  200-Atm  pixel  pitch.  (The  pixel  size  is  primarily 
determined  by  coarse  metal  design  rules,  since  the  host  device 
on  the  wafer  is  a  back-side  illuminated,  thinned  visible  im¬ 
ager.)  Each  pixel  consists  of  two  photo-capacitors,  two  stages 
of  digital  shift  register,  and  the  associated  circuitry  to  perform 
ternary  weighting. 

The  results  shown  are  for  a  serial  output  register  rate  of  1.0 
MHz.  The  various  Tl,  T2,  T3,  T4  portions  of  the  cycle  are 
being  run  non-overlapped  to  simplify  the  operating  electronics 
requirements.  The  resulting  cycle  rate  is  2500  cycles/s.  In  this 
operating  mode,  8  x  10^  differentially  sensed  outer-product 
terms  are  produced  per  second. 

One  of  the  advantages  of  the  transversal  imager  architecture 
is  that  the  operating  cycle  does  not  change  if  the  operating 
mode  is  changed.  The  same  operating  cycle  is  utilized  in  the 
optical  state  vector  and  electrical  synaptic  weights  mode  as 
is  used  in  the  optical  synaptic  weights  and  electrical  state 
vector  mode.  The  transversal  imager  therefore  sustains  the 
same  outer-product  rate  in  either  mode. 

In  the  results  presented  below,  the  actual  photocharge 
weighting  and  sununation  in  the  charge  domain  is  completed 
in  parallel  across  the  complete  array  in  a  time  T2  =  20  /xs. 
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Fig.  4.  Final  outer-product  result  for  uniform  optical  vector  input  and  ternary 
synaptic  weights. 


Fig.  5.  Final  outer-product  result  for  linearly  increasing  optical  vector  input 
and  uniform  negative  synaptic  weights. 


This  time  is  dominated  by  the  photocharge  thermal  diffusion 
transfer  time,  which  is  long  given  the  large  pixels.  In  this 
20  /iS  period  2000  analog-binary  multiplications  and  2000 
additions  are  performed. 

The  preliminary  results  shown  in  Fig.  4  show  the  final 
differential  output  of  18  columns  for  a  uniform  light  input 

and  a  complete  column  weighting  of  (left-to-right):  4 - ^ — 

-fOOOO  0000- +  H - .  (A  portion  of  the  small  non-uniformity 

in  response  is  due  to  non-uniform  illumination.)  Fig.  5  shows 
the  final  transversal  imager  output  for  a  uniform  negative 
weight  and  a  wedge-shape  optical  input  pattern. 

VIL  Conclusion 

A  novel  electro-optical  neural  network  processor  has  been 
developed.  The  processor  is  based  on  MOS/CCD  integrated 
circuit  technology.  The  32  x  32  transversal  imager  demon¬ 
strated  performs  ternary  weighting.  This  first  implementation 


utilizing  a  200  x  200- fim  cell  pitch,  1.5-MHz  clocking,  and 
a  non-overlapping  operating  cycle  achieves  5  000  000  analog 
binary  operations/s  (2  500  000  multiplications,  and  2  500  000 
additions).  As  a  neural  network  processor  the  current  im¬ 
plementation  achieves  32  fully  interconnected  neurons  with 
an  interconnection  update  rate  of  2.5  x  10^  interconnection 
updates/s. 

This  architecture,  the  overlapped  operating  cycle  discussed, 
and  today's  silicon  IC  technology  would  permit  scaling  to 
a  75  X  75-^im  cell  pitch  and  10-MHz  clocking.  This  would 
permit  realization  of  a  128  x  128  processor  with  128  fully 
interconnected  neurons  processing  at  an  update  rate  of  7  x 
10^  interconnection  updates  per  second.  Furthermore,  the 
architecture  permits,  even  at  these  high  processing  rates,  the 
optical  input  and  the  electrical  input  to  both  be  updated  during 
each  cycle.  This  would  be  very  useful  for  learning  algorithms 
where  it  is  desirable  to  update  the  synaptic  weights  frequently. 

The  architecture  permits  extension  to  larger  neural  networks 
by  tiling  of  the  transversal  imager  devices  in2x2or4x4 
mosaics.  In  the  configuration  with  optical  input  of  the  state 
vector,  to  achieve  these  larger  networks,  an  optical  input  device 
with  twice  or  four  times  the  number  of  elements  would  be 
required.  The  optical  input  would  be  spread  across  the  rows 
of  the  mosaic  and  the  column  partial  outer-product  outputs 
from  each  device  would  be  combined  within  the  columns 
of  the  mosaic  to  produce  the  desired  outer-product  result. 
The  transversal  imager  devices  would  be  run  in  parallel  with 
common  clocking  signals.  In  this  manner  the  architecture 
can  be  extended  to  realization  of  larger  electro-optical  neural 
processors. 
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